Kikuchi-Bayes: Factorized Models for Approximate Classification in Closed Form

We propose a simple family of classification models, based on the Kikuchi approximation to free energy, that generalize upon the naive Bayesian classifier. The resulting product of potentials is not normalized, but for classification it is easy to perform the normalization for each instance separately, just as in naive Bayes. Our learning algorithm creates the set of initial regions by including only those initial regions that provide a significant improvement to the approximation which does not include them. We observe that this algorithm outperforms other methods, such as the tree-augmented naive Bayes, but that the inclusion of regions may increase the approximation error. For that reason we recommend separating the generalization error which arises from the mismatch between the training and test data, and approximation error which arises because of imperfect model. The approximation error was the dominant source of variation in experiments we performed on realistic data sets.